End-to-end Memory-Efficient Reconstruction for Cone Beam CT

Cone Beam CT plays an important role in many medical fields nowadays, but the potential of this imaging modality is hampered by lower image quality compared to the conventional CT. A lot of recent research has been directed towards reconstruction methods relying on deep learning. However, practical application of deep learning to CBCT reconstruction is complicated by several issues, such as exceedingly high memory costs of deep learning methods for fully 3D data. In this work, we address these limitations and propose LIRE: a learned invertible primal-dual iterative scheme for Cone Beam CT reconstruction. Memory requirements of the network are substantially reduced while preserving its expressive power, enabling us to train on data with isotropic 2mm voxel spacing, clinically-relevant projection count and detector panel resolution on current hardware with 24 GB VRAM. Two LIRE models for small and for large Field-of-View setting were trained and validated on a set of 260 + 22 thorax CT scans and tested using a set of 142 thorax CT scans plus an out-of-distribution dataset of 79 head \& neck CT scans. For both settings, our method surpasses the classical methods and the deep learning baselines on both test sets. On the thorax CT set, our method achieves PSNR of 33.84 $\pm$ 2.28 for the small FoV setting and 35.14 $\pm$ 2.69 for the large FoV setting; U-Net baseline achieves PSNR of 33.08 $\pm$ 1.75 and 34.29 $\pm$ 2.71 respectively. On the head \& neck CT set, our method achieves PSNR of 39.35 $\pm$ 1.75 for the small FoV setting and 41.21 $\pm$ 1.41 for the large FoV setting; U-Net baseline achieves PSNR of 33.08 $\pm$ 1.75 and 34.29 $\pm$ 2.71 respectively. Additionally, we demonstrate that LIRE can be finetuned to reconstruct high-resolution CBCT data with the same geometry but 1mm voxel spacing and higher detector panel resolution, where it outperforms the U-Net baseline as well

Deep Cardiac MRI Reconstruction with ADMM

Cardiac magnetic resonance imaging is a valuable non-invasive tool for identifying cardiovascular diseases. For instance, Cine MRI is the benchmark modality for assessing the cardiac function and anatomy. On the other hand, multi-contrast (T1 and T2) mapping has the potential to assess pathologies and abnormalities in the myocardium and interstitium. However, voluntary breath-holding and often arrhythmia, in combination with MRI's slow imaging speed, can lead to motion artifacts, hindering real-time acquisition image quality. Although performing accelerated acquisitions can facilitate dynamic imaging, it induces aliasing, causing low reconstructed image quality in Cine MRI and inaccurate T1 and T2 mapping estimation. In this work, inspired by related work in accelerated MRI reconstruction, we present a deep learning (DL)-based method for accelerated cine and multi-contrast reconstruction in the context of dynamic cardiac imaging. We formulate the reconstruction problem as a least squares regularized optimization task, and employ vSHARP, a state-of-the-art DL-based inverse problem solver, which incorporates half-quadratic variable splitting and the alternating direction method of multipliers with neural networks. We treat the problem in two setups; a 2D reconstruction and a 2D dynamic reconstruction task, and employ 2D and 3D deep learning networks, respectively. Our method optimizes in both the image and k-space domains, allowing for high reconstruction fidelity. Although the target data is undersampled with a Cartesian equispaced scheme, we train our model using both Cartesian and simulated non-Cartesian undersampling schemes to enhance generalization of the model to unseen data. Furthermore, our model adopts a deep neural network to learn and refine the sensitivity maps of multi-coil k-space data. Lastly, our method is jointly trained on both, undersampled cine and multi-contrast data.Comment: 12 pages, 3 figures, 2 tables. CMRxRecon Challenge, MICCAI 202

On Retrospective k-space Subsampling schemes For Deep MRI Reconstruction

Purpose: Acquiring fully-sampled MRI $k$-space data is time-consuming, and collecting accelerated data can reduce the acquisition time. Employing 2D Cartesian-rectilinear subsampling schemes is a conventional approach for accelerated acquisitions; however, this often results in imprecise reconstructions, even with the use of Deep Learning (DL), especially at high acceleration factors. Non-rectilinear or non-Cartesian trajectories can be implemented in MRI scanners as alternative subsampling options. This work investigates the impact of the $k$-space subsampling scheme on the quality of reconstructed accelerated MRI measurements produced by trained DL models. Methods: The Recurrent Variational Network (RecurrentVarNet) was used as the DL-based MRI-reconstruction architecture. Cartesian, fully-sampled multi-coil $k$-space measurements from three datasets were retrospectively subsampled with different accelerations using eight distinct subsampling schemes: four Cartesian-rectilinear, two Cartesian non-rectilinear, and two non-Cartesian. Experiments were conducted in two frameworks: scheme-specific, where a distinct model was trained and evaluated for each dataset-subsampling scheme pair, and multi-scheme, where for each dataset a single model was trained on data randomly subsampled by any of the eight schemes and evaluated on data subsampled by all schemes. Results: In both frameworks, RecurrentVarNets trained and evaluated on non-rectilinearly subsampled data demonstrated superior performance, particularly for high accelerations. In the multi-scheme setting, reconstruction performance on rectilinearly subsampled data improved when compared to the scheme-specific experiments. Conclusion: Our findings demonstrate the potential for using DL-based methods, trained on non-rectilinearly subsampled measurements, to optimize scan time and image quality.Comment: 24 pages, 12 figures, 5 table

A multi-resolution strategy for a multi-objective deformable image registration framework that accommodates large anatomical differences

Currently, two major challenges dominate the field of deformable image registration. The first challenge is related to the tuning of the developed methods to specific problems (i.e. how to best combine different objectives such as similarity measure and transformation effort). This is one of the reasons why, despite significant progress, clinical implementation of such techniques has proven to be difficult. The second challenge is to account for large anatomical differences (e.g. large deformations, (dis)appearing structures) that occurred between image acquisitions. In this paper, we study a framework based on multi-objective optimization to improve registration robustness and to simplify tuning for specific applications. Within this framework we specifically consider the use of an advanced model-based evolutionary algorithm for optimization and a dual-dynamic transformation model (i.e. two "non-fixed" grids: one for the source- and one for the target image) to accommodate for large anatomical differences. The framework computes and presents multiple outcomes that represent efficient trade-offs between the different objectives (a so-called Pareto front). In image processing it is common practice, for reasons of robustness and accuracy, to use a multi-resolution strategy. This is, however, only well-established for single-objective registration methods. Here we describe how such a strategy can be realized for our multi-objective approach and compare its results with a single-resolution strategy. For this study we selected the case of prone-supine breast MRI registration. Results show that the well-known advantages of a multi-resolution strategy are successfully transferred to our multi-objective approach, resulting in superior (i.e. Pareto-dominating) outcomes

Few-shot Semantic Segmentation with Support-induced Graph Convolutional Network

Few-shot semantic segmentation (FSS) aims to achieve novel objects segmentation with only a few annotated samples and has made great progress recently. Most of the existing FSS models focus on the feature matching between support and query to tackle FSS. However, the appearance variations between objects from the same category could be extremely large, leading to unreliable feature matching and query mask prediction. To this end, we propose a Support-induced Graph Convolutional Network (SiGCN) to explicitly excavate latent context structure in query images. Specifically, we propose a Support-induced Graph Reasoning (SiGR) module to capture salient query object parts at different semantic levels with a Support-induced GCN. Furthermore, an instance association (IA) module is designed to capture high-order instance context from both support and query instances. By integrating the proposed two modules, SiGCN can learn rich query context representation, and thus being more robust to appearance variations. Extensive experiments on PASCAL-5i and COCO-20i demonstrate that our SiGCN achieves state-of-the-art performance.Comment: Accepted in BMVC2022 as oral presentatio

A first step toward uncovering the truth about weight tuning in deformable image registration

Deformable image registration is currently predominantly solved by optimizing a weighted linear combination of objectives. Successfully tuning the weights associated with these objectives is not trivial, leading to trial-and-error approaches. Such an approach assumes an intuitive interplay between weights, optimization objectives, and target registration errors. However, it is not known whether this always holds for existing registration methods. To investigate the interplay between weights, optimization objectives, and registration errors, we employ multi-objective optimization. Here, objectives of interest are optimized simultaneously, causing a set of multiple optimal solutions to exist, called the optimal Pareto front. Our medical application is in breast cancer and includes the challenging prone-supine registration problem. In total, we studied the interplay in three different ways. First, we ran many random linear combinations of objectives using the well-known registration software elastix. Second, since the optimization algorithms used in registration are typically of a local-search nature, final solutions may not always form a Pareto front. We therefore employed a multi-objective evolutionary algorithm that finds weights that correspond to registration outcomes that do form a Pareto front. Third, we examined how the interplay differs if a true multi-objective (i.e., weight-free) image registration method is used. Results indicate that a trial-and-error weight-adaptation approach can be successful for the easy prone to prone breast image registration case, due to the absence of many local optima. With increasing problem difficulty the use of more advanced approaches can be of value in finding and selecting the optimal registration outcomes

On the usefulness of gradient information in multi-objective deformable image registration using a B-spline-based dual-dynamic transformation model: comparison of three optimization algorithms

The use of gradient information is well-known to be highly useful in single-objective optimization-based image registration methods. However, its usefulness has not yet been investigated for deformable image registration from a multi-objective optimization perspective. To this end, within a previously introduced multi-objective optimization framework, we use a smooth B-spline-based dual-dynamic transformation model that allows us to derive gradient information analytically, while still being able to account for large deformations. Within the multi-objective framework, we previously employed a powerful evolutionary algorithm (EA) that computes and advances multiple outcomes at once, resulting in a set of solutions (a so-called Pareto front) that represents efficient trade-offs between the objectives. With the addition of the B-spline-based transformation model, we studied the usefulness of gradient information in multiobjective deformable image registration using three different optimization algorithms: the (gradient-less) EA, a gradientonly algorithm, and a hybridization of these two. We evaluated the algorithms to register highly deformed images: 2D MRI slices of the breast in prone and supine positions. Results demonstrate that gradient-based multi-objective optimization significantly speeds up optimization in the initial stages of optimization. However, allowing sufficient computational resources, better results could still be obtained with the EA. Ultimately, the hybrid EA found the best overall approximation of the optimal Pareto front, further indicating that adding gradient-based optimization for multiobjective optimization-based deformable image registration can indeed be beneficial

Effects of anatomical changes on pencil beam scanning proton plans in locally advanced NSCLC patients

Daily anatomical variations can cause considerable differences between delivered and planned dose. This study simulates and evaluates these effects in spot-scanning proton therapy for lung cancer patients. Robust intensity modulated treatment plans were designed on the mid-position CT scan for sixteen locally advanced lung cancer patients. To estimate dosimetric uncertainty, deformable registration was performed on their daily CBCTs to generate 4DCT equivalent scans for each fraction and to map recomputed dose to a common frame. Without adaptive planning, eight patients had an undercoverage of the targets of more than 2GyE (maximum of 14.1GyE) on the recalculated treatment dose from the daily anatomy variations including respiration. In organs at risk, a maximum increase of 4.7GyE in the D1 was found in the mediastinal structures. The effect of respiratory motion alone is smaller: 1.4GyE undercoverage for targets and less than 1GyE for organs at risk. Daily anatomical variations over the course of treatment can cause considerable dose differences in the robust planned dose distribution. An advanced planning strategy including knowledge of anatomical uncertainties would be recommended to improve plan robustness against interfractional variations. For large anatomical changes, adaptive therapy is mandator

Diversifying Multi-Objective Gradient Techniques and their Role in Hybrid Multi-Objective Evolutionary Algorithms for Deformable Medical Image Registration

Gradient methods and their value in single-objective, real-valued optimization are well-established. As such, they play a key role in tackling real-world, hard optimization problems such as deformable image registration (DIR). A key question is to which extent gradient techniques can also play a role in a multi-objective approach to DIR. We therefore aim to exploit gradient information within an evolutionary-algorithm-based multi-objective optimization framework for DIR. Although an analytical description of the multi-objective gradient (the set of all Pareto-optimal improving directions) is available, it is nontrivial how to best choose the most appropriate direction per solution because these directions are not necessarily uniformly distributed in objective space. To address this, we employ a Monte-Carlo method to obtain a discrete, spatially-uniformly distributed approximation of the set of Pareto-optimal improving directions. We then apply a diversification technique in which each solution is associated with a unique direction from this set based on its multi- as well as single-objective rank. To assess its utility, we compare a state-of-the-art multi-objective evolutionary algorithm with three different hybrid versions thereof on several benchmark problems and two medical DIR problems. Results show that the diversification strategy successfully leads to unbiased improvement, helping an adaptive hybrid scheme solve all problems, but the evolutionary algorithm remains the most powerful optimization method, providing the best balance between proximity and diversity